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Three  Classes  of  Appraisals 


Characteristic 

Class  C 

Class  B 

Class  A 

Amount  of  objective  evidence 

Low 

Medium 

High 

Ratings  generated 

No 

No 

Yes 

Resource  needs 

Low 

Medium 

High 

Team  Size 

Small 

Medium 

Large 
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Are  We  Talking  About  the  Same  Thing? 


“Mini  Assessment” 
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The  SEI SCAMPISM  B&C  Project 

Develop,  Field  Test,  and  Deploy  an  integrated  suite  of  appraisal 
methods  that  are: 

•  Developed  iteratively  with  extensive  field  tests 

•  Based  on  requirements  from  users 

•  Integrated  and  upwardly  compatible 

•  Well  specified  and  documented 
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Presentation  Outline 

Examples  of  Appraisal  Types 
Motivation  and  Purpose  of  Appraisals 
Data  Collection  Techniques  Used 
Three  Areas  of  Specification 
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Getting-Started  Intervention 

Purpose:  Provide  information  to  an 
organization  learning  about 
CMMISM. 

Description:  Expert-directed, 
presentation  &  discussion¬ 
intensive,  broad  scope,  shallow 
coverage  method  that  emphasizes 
participation  of  affected  groups  in 
facilitated  sessions  -  with  no  focus 
on  sufficiency  of  coverage. 

Rough  Effort  Estimate:  One  to  two 
experts  on  site  for  1  to  5  days. 

Planning  and  coordination  effort 
driven  by  appraisal  scope. 
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Mini  Appraisal 

Purpose:  An  abbreviated  version  of 
a  benchmarking  appraisal  used  for 
experience  and  non-rating  related 
outputs. 

Description:  Team-based,  data 
intensive,  narrow  scope,  deep 
coverage  method  that  emphasizes 
the  use  of  Objective  Evidence  and 
interviews  -  with  reduced  focus  on 
data  sufficiency. 

Rough  Effort  Estimate:  One  expert 
on  site  for  3  to  5  days.  Planning 
and  coordination  effort  driven  by 
appraisal  scope,  and  experience 
level  of  local  team  members. 
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Gap  Analysis 

Purpose:  Identify  specific 
deficiencies  in  implemented 
practices  relative  to  practices  which 
would  support  achievement  of  the 
goals  -  for  a  specified  scope. 

Description:  Expert-based, 
interview-intensive,  broad  scope, 
shallow  coverage  method  that 
emphasizes  identification  of 
weaknesses  -  with  more  limited 
focus  on  sufficiency  of  coverage. 

Rough  Effort  Estimate:  One  to  two 
experts  on  site  for  1  to  3  days. 
Planning  and  coordination  effort 
driven  by  appraisal  scope. 


©  2003  by  Carnegie  Mellon  University 


wh  SEPG03 


page  8 


Carnegie  Mellon 

Software  Engineering  Institute 


Improvement  Monitoring 

Purpose:  Track  implementation  of 
process  improvement  actions  in  an 
organizational  unit  that  has 
previously  undergone  a 
benchmarking  appraisal. 

Description:  Expert-based, 
interview  &  document-intensive, 
narrow  scope,  deep  coverage 
method  that  emphasizes  identifying 
status  of  changes  in  processes  - 
with  more  limited  focus  on 
sufficiency  of  coverage. 

Rough  Effort  Estimate:  One  to  two 
experts  on  site  for  1  to  3  days. 

Planning  and  coordination  effort 
driven  by  scope  and  outcomes  of 
previous  appraisal(s). 
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Delta  Appraisal 

Purpose:  Confirm  corrective  actions 
resulting  from  a  benchmarking 
appraisal. 

Description:  Expert-based, 
interview  &  document-intensive, 
very  narrow  scope,  deep  coverage 
method  that  emphasizes  confirming 
specific  changes  in  processes  - 
with  limited  focus  on  sufficiency  of 
coverage  for  the  model. 

Rough  Effort  Estimate:  One  to  two 
experts  on  site  for  1  or  2  days. 
Planning  and  coordination  effort 
driven  by  the  nature  and  magnitude 
of  weaknesses  identified  in  the 
benchmarking  appraisal. 
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Incremental  Appraisal 

Purpose:  Building  a  profile  of 
results  across  a  broad  model 
and/or  organizational  scope. 

Description:  Expert-directed  or 
team-based,  data  intensive, 
narrow  scope,  deep  coverage 
method  that  emphasizes 
use  of  objective  evidence  -  with 
strong  focus  on  sufficiency  of 
coverage  (in  narrow  increments). 

Rough  Effort  Estimate:  Appraisal 
events  distributed  over  time.  One 
expert  on  site  for  3  to  8  days. 

Planning  and  coordination  effort 
driven  by  appraisal  scope,  and 
experience  level  of  team 
members. 


©  2003  by  Carnegie  Mellon  University 


wh  SEPG03 


page  11 


Carnegie  Mellon 

Software  Engineering  Institute 


Benchmarking  Appraisal 

Purpose:  Document  a  frequently 
sought,  high  level,  benchmark  for 
an  organizational  unit. 

Description:  Expert-directed  and 
team-based,  data  intensive,  broad 
scope,  deep  coverage  method  that 
emphasizes  use  of  objective 
evidence  -  with  strong  focus  on 
sufficiency  of  coverage. 

Rough  Effort  Estimate:  One  expert 
on  site  for  5  to  15  days.  Planning 
and  coordination  effort  driven  by 
appraisal  scope,  organizational 
maturity,  and  experience  level  of 
team  members. 
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Presentation  Outline 

Examples  of  Appraisal  Types 
Motivation  and  Purpose  of  Appraisals 
Data  Collection  Techniques  Used 
Three  Areas  of  Specification 
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Motivation  for  Appraisal 


Technical  Analysis 

•  Evaluating  detailed  practices 

•  Rigorous  data  collection 

•  Carefully  defined  scope 

•  Potential  external  visibility 


Organizational  Intervention 

•  Assessing  culture  for  change 

•  Building  local  championship 

•  Preparing  for  improvement 

•  Finding  the  stimulus  for  change 
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Purpose  of  Appraisal  -1 


Formulating  Our 
Winning  Strategy 


Exploring  Goals  for 
Improvement  & 
Training  Staff 


Fixing  Immediate 
Problems 
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Purpose  of  Appraisal  -2 


Evaluating  and 


Monitoring  Risk 
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Examples  of  Appraisal  Types 
Motivation  and  Purpose  of  Appraisals 
Data  Collection  Techniques  Used 
Three  Areas  of  Specification 
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Traditional  Data  Collection  Techniques 
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Evolution  of  Data  Collection 
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Data  Collection  Techniques  Used 
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Requirements  Based  Tailoring 
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Sampling  Participants 


Choosing  who  to  involve  in 
specific  data  collection 
activities  requires  one  to 
make  sampling  decisions. 

Preserving  a  coherent 
scope  for  the  appraisal 
requires  specification  of 
meaningful  sampling 
criteria. 


r  ivision  XV. 


Role  A1 


Role  A3 
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Sampling: 

Broad  Scope,  Shallow  Coverage 

Take  advantage  of  low  cost  data  collection  methods 

•  achieve  broader  coverage  of  the  organization 

-  include  all  projects,  not  just  4  for  benchmarking 

•  address  a  larger  portion  of  the  model 

-  match  information  sources  to  information  needs 

Limitations  of  low  cost  data  collection  methods 

•  very  limited  opportunity  to  explain  nuances 

-  yes/no  questions  in  interviews  don’t  pay  off 

•  difficult  to  assure  validity  of  data 

-  ambiguous  questionnaire  items  don’t  help 
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Sampling: 

Narrow  Scope,  Deep  Coverage 

Maximize  insight  with  a  small  sample  of  the  organization 

•  detailed  insight  about  a  selected  unit  in  the  organization 

-  potentially  include  every  staff  member 

•  full  coverage  of  limited  model  material 

-  minimal  risk  of  undiscovered  weaknesses 

Limitations  of  using  a  small  sample 

•  limited  ability  to  make  generalizations 

-  process  of  one  stellar  project  may  be  misleading 

•  limited  coverage  of  the  lifecycle  in  use 

-  implementation  of  future  phases  may  differ  from 
intended  practices 
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Summarizing  and  Presenting  Results 


Statements  of  strengths 
and/or  weaknesses  are 
most  common. 

Characterizations  of 
differences  in  practice 
implementation 

•  across  model  content 

•  across  the  organization 

Profiles  of  detailed  data,  or 
prioritized  lists  of  statements 
are  also  common. 


Your  Results 

•  Findings  statements 

•  Strengths/Weaknesses 

•  Recommendations 
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The  SCAMPISM  B&C  Project 

SEI  working  with  appraisal  stakeholders: 

•  prototyping  and  field  trials  with  collaborator  organizations 

•  communication  with  stakeholders  about  the  project 

•  periodic  dissemination  of  draft  material  for  review 

•  publication/presentation  of  project  results 


Points  of  Contact 

Will  Hayes  wh@sei.cmu.edu 

Gene  Miluk  gem@sei.cmu.edu 
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